fix(actions): keyboard-occlusion guard Phase 1 — inject hideKeyboard in Maestro generator (#356) by Lykhoyda · Pull Request #367 · Lykhoyda/rn-dev-agent

Lykhoyda · 2026-06-20T21:33:10Z

Closes #356 (Phase 1).

Problem

When a learned-action replay taps a bottom-pinned control (submit/continue) while the soft keyboard is up, the tap lands on the keyboard instead of the button, so the next screen is never reached. The #300 session flagged this as the single biggest source of flaky replays, on both iOS and Android. The reporter's proven manual workaround was - hideKeyboard before each such tap.

Why this is Phase 1 (and what it is NOT)

The bug spans two independent device-control engines that do not share a tap path:

Surface	Path	This PR?
Learned-action replay (`cdp_run_action` → `maestro_run`)	external maestro-runner (WDA/UIAutomator) — never hits `runNative()`	✅ Phase 1
Live `device_press` / `device_batch`	`runNative()` → in-tree runner	⬜ Phase 2 (separate PR)

A guard in our native runner is a no-op for action replays (they shell out to the external maestro-runner). So Phase 1 fixes the L3/Maestro replay path — the headline symptom. The L2/in-runner frame-precise guard for live taps is deferred to Phase 2 (its design decisions are recorded in the spec).

Approach

A pure, order-only transform inside generateMaestro() — the single chokepoint every generated/saved Maestro flow passes through. It tracks one boolean (keyboardLikelyUp): a type event raises it; before a tap/long_press it emits a - hideKeyboard step (with an audit comment) when up, then lowers it; navigate resets it; submit does not.

Safe by construction:

hideKeyboard is already allowlisted (maestro-validator) and is a no-op when no keyboard is showing, so over-injection costs only a few hundred ms.
Maestro taps are selector-based, so after dismiss Maestro re-resolves the element's position — zero stale-coordinate risk.

Emitted shape

- tapOn:
    id: emailInput
- inputText: "user@example.com"
# rn-dev-agent: keyboard-occlusion guard (#356)
- hideKeyboard
- tapOn:
    id: submitButton

Scope

In: generation-time injection in generateMaestro (covers cdp_record_test_generate + save-as-action).
Out (deferred): Phase 2 in-runner live-tap guard; Phase 1.5 repair-time injection + existing-corpus backfill; Detox parity; swipe/fill occlusion.

Testing

Unit: 7 new table-driven cases over event sequences (typed→tap injects; no-type→no inject; single injection across two taps; navigate resets; consecutive types; submit no-reset; long_press guarded). Full suite 2424/2424 passing.
Reviews: multi-LLM plan review (Codex+Gemini) pre-code, per-task spec+quality reviews, and a final whole-branch review (which caught a tracked-dist/ omission, now fixed).

⏳ Remaining pre-merge gate

On-device verification on iOS + Android (per the issue): record a fill→bottom-pinned-submit flow, replay, confirm the submit reliably reaches the next screen where it was previously flaky. Not yet run (environment setup) — will complete before merge.

Docs

Spec + TDD plan committed under docs/superpowers/{specs,plans}/2026-06-20-356-*.

🤖 Generated with Claude Code

…deKeyboard injection) Generation-time hideKeyboard injection in generateMaestro() to fix flaky action replays where bottom-pinned taps land on the soft keyboard. Phase 1 covers the L3/Maestro replay path (cdp_run_action -> maestro-runner); the L2/in-runner live-tap guard is deferred to Phase 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…tro generator) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dist/ is tracked and loaded by the runtime MCP bridge; the src change in prior commits was inert in installs until this rebuilt artifact ships. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47bf54fa0f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-20T21:35:48Z

+        if (sel) {
+          if (keyboardLikelyUp) {
+            lines.push('# rn-dev-agent: keyboard-occlusion guard (#356)');
+            lines.push('- hideKeyboard');


Support hideKeyboard in the CDP replay fallback

When a generated action includes this new - hideKeyboard step, cdp_run_action can no longer use its iOS 26.x CDP/JS fallback for the same fill→submit flows: runCdpReplay() normalizes the saved YAML through normalizeSteps, which only special-cases the string waitForAnimationToEnd and otherwise throws UnsupportedStepError for string commands such as hideKeyboard (scripts/cdp-bridge/src/domain/cdp-flow-replay.ts:38-46). In the known transport-blind path where Maestro/WDA reports SELECTOR_NOT_FOUND or UNKNOWN but the selector is present in CDP, actions generated here will now return UNSUPPORTED_STEP instead of replaying; please either map hideKeyboard in the CDP replay subset or avoid emitting it for flows that may need that fallback.

Useful? React with 👍 / 👎.

Lykhoyda · 2026-06-20T21:56:39Z

✅ Device verification complete (iOS + Android) — with one runtime finding

Ran the injected guard sequence on real devices using the wizard screen (wizard-title-input → wizard-next-btn, a bottom-pinned next button behind the keyboard).

Path	Result
iOS via maestro-runner (plugin default)	✅ Full flow PASS 8/8 — `hideKeyboard` dismissed (309ms), `tapOn wizard-next-btn` landed, advanced to STEP 2 OF 3
Android via official Maestro v2.3.0	✅ PASS — keyboard up → `hideKeyboard` dismissed (`mInputShown=false`) → next tap landed → STEP 2 OF 3
Android via maestro-runner (plugin default)	❌ `hideKeyboard` is a no-op — reports PASS in 5ms but `mInputShown=true` after; next button stays occluded

Finding (filed as B223 → will open a follow-up issue)

The generator fix in this PR is correct — it emits the right YAML, verified on iOS and on Android via official Maestro. But the plugin's default L3 replay engine, the third-party maestro-runner (DeviceLab.dev 1.0.9), silently no-ops hideKeyboard on Android. Isolated cleanly:

adb shell input keyevent KEYCODE_BACK → keyboard dismisses (mInputShown=false) ✓
official Maestro hideKeyboard → dismisses ✓
maestro-runner hideKeyboard → does not dismiss ✗ (5ms, mInputShown=true)

Net: iOS is fully fixed via the default path. Android is fixed via official Maestro, but not via the default maestro-runner until that runner bug is addressed.

Recommended follow-up (separate PR): in maestro_run, when a flow contains hideKeyboard AND platform=android, prefer the official maestro CLI; and report the no-op upstream to maestro-runner. (Details + repro in B223.)

Proof artifacts: docs/proof/356-keyboard-occlusion/ (ios-wizard-step2.jpg, android-wizard-step2-official.png).

…223) maestro-runner v1.0.9 silently no-ops hideKeyboard on Android (reports pass in ~5ms, mInputShown stays true), defeating the keyboard-occlusion guard. chooseMaestroDispatch now prefers the official Maestro CLI for Android flows that contain hideKeyboard (verified to honor it on-device), and surfaces a degradedReason warning when the CLI is unavailable. iOS is unaffected (maestro-runner honors hideKeyboard there). Device-verified: official Maestro dismisses the Android keyboard and the bottom-pinned tap lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…+ changeset Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Lykhoyda · 2026-06-21T21:08:28Z

Update: Android `hideKeyboard` runner-routing added to this PR

Per the verification finding above, I've added the Android fix to this PR (rather than deferring it):

chooseMaestroDispatch now takes a flowHasHideKeyboard input; on Android, a flow containing hideKeyboard routes to the official Maestro CLI (verified on-device to dismiss the keyboard), falling back to maestro-runner with a degradedReason warning when the CLI isn't installed. iOS is unchanged (maestro-runner honors hideKeyboard there).
maestro_run detects hideKeyboard in the parsed commands (flowContainsHideKeyboard) and passes the flag; dispatch is now chosen after flow parsing.
5 new TDD dispatch tests covering all four routing cases + the helper. Full suite 2429/2429. dist/ committed.
Reviewed (spec ✅; one Minor JSDoc-accuracy nit fixed).

Net result: Phase 1 now fixes the keyboard-occlusion replay bug on both platforms via the plugin's default path — iOS via maestro-runner, Android via the auto-selected official Maestro CLI. B223 (the underlying maestro-runner no-op) remains a tracked upstream/runtime follow-up.

Decisions logged: D1286. Spec + changeset updated.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 525ac0fbe7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-21T21:11:41Z

+  return commands.some(
+    (c) =>
+      c === 'hideKeyboard' ||
+      (typeof c === 'object' && c !== null && 'hideKeyboard' in (c as Record<string, unknown>)),
+  );


Recurse into runFlow commands for hideKeyboard

When a validated Android flow hides the keyboard inside an inline or conditional runFlow, for example { runFlow: { when, commands: ['hideKeyboard', ...] } }, this shallow some returns false because it only checks the root command object. parseAndValidateFlow() preserves those nested commands for inline/conditional runFlow cases (see scripts/cdp-bridge/src/domain/maestro-validator.ts:387-399), so the flow is routed to maestro-runner and Android hideKeyboard is no-oped; please recurse into runFlow.commands when computing flowHasHideKeyboard.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-21T21:11:41Z

+  // Android (`mInputShown=false` after). iOS maestro-runner honors hideKeyboard,
+  // so this only applies to Android.
+  const needsOfficialForKeyboard =
+    inputs.platform === 'android' && inputs.flowHasHideKeyboard === true;


Route hideKeyboard flows outside maestro_run

When an Android flow containing the newly emitted top-level - hideKeyboard is run through other supported flow runners, those paths still call chooseMaestroDispatch({ platform }) without the new flag (scripts/cdp-bridge/src/tools/maestro-test-all.ts:77 and scripts/cdp-bridge/src/maestro-invoke.ts:60), so this strict check stays false and the dispatcher selects maestro-runner—the Android path the new comments say no-ops hideKeyboard. Please have every flow-running path pass the parsed flag, or move detection into dispatch, otherwise Android regression suites and inline Maestro flows do not get the keyboard-occlusion fix.

Useful? React with 👍 / 👎.

…223) The batch replay surface (maestro_test_all) chose dispatch once before its per-flow loop, so Android replays of saved actions containing hideKeyboard still hit maestro-runner's no-op. Now re-routes per flow to the official Maestro CLI when a flow uses hideKeyboard on Android, surfacing a degraded warning if the CLI is unavailable. Mirrors the maestro_run routing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Lykhoyda · 2026-06-21T21:16:29Z

Final whole-branch review (opus) flagged that maestro_test_all (batch replay of saved actions) chose its runner once before the per-flow loop — so Android batch replays of hideKeyboard flows would still hit maestro-runner's no-op. Fixed in c95f0d3: it now re-routes per flow to the official Maestro CLI when a flow uses hideKeyboard on Android (mirroring maestro_run), with the same degraded-warning fallback. Full suite 2429/2429; lint+format clean; dist committed. Both action-replay surfaces (maestro_run and maestro_test_all) are now covered on Android.

Lykhoyda and others added 6 commits June 20, 2026 23:08

docs(356): TDD implementation plan — keyboard-occlusion guard Phase 1

cf930cb

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(356): inject hideKeyboard before taps following text entry (Maes…

3e37959

…tro generator) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(356): reset keyboard guard on navigate; guard long_press too

160bb78

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chore(changeset): keyboard-occlusion guard Phase 1 (cdp + plugin patch)

67f420f

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

build(356): stage rebuilt dist for hideKeyboard injection

47bf54f

dist/ is tracked and loaded by the runtime MCP bridge; the src change in prior commits was inert in installs until this rebuilt artifact ships. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chatgpt-codex-connector Bot reviewed Jun 20, 2026

View reviewed changes

Lykhoyda and others added 2 commits June 21, 2026 23:05

docs(356): record Android hideKeyboard runner-routing (B223) in spec …

525ac0f

…+ changeset Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chatgpt-codex-connector Bot reviewed Jun 21, 2026

View reviewed changes

Lykhoyda merged commit d61985f into main Jun 21, 2026
11 checks passed

Lykhoyda deleted the feat/356-keyboard-occlusion-guard-phase1 branch June 21, 2026 21:19

Lykhoyda mentioned this pull request Jul 1, 2026

feat(keyboard-guard): in-runner keyboard-occlusion guard for live device_* taps, iOS + Android (#370) #380

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(actions): keyboard-occlusion guard Phase 1 — inject hideKeyboard in Maestro generator (#356)#367

fix(actions): keyboard-occlusion guard Phase 1 — inject hideKeyboard in Maestro generator (#356)#367
Lykhoyda merged 9 commits into
mainfrom
feat/356-keyboard-occlusion-guard-phase1

Lykhoyda commented Jun 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 20, 2026

Uh oh!

Lykhoyda commented Jun 20, 2026

Uh oh!

Lykhoyda commented Jun 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 21, 2026

Uh oh!

chatgpt-codex-connector Bot Jun 21, 2026

Uh oh!

Lykhoyda commented Jun 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Lykhoyda commented Jun 20, 2026

Problem

Why this is Phase 1 (and what it is NOT)

Approach

Emitted shape

Scope

Testing

⏳ Remaining pre-merge gate

Docs

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 20, 2026

Choose a reason for hiding this comment

Uh oh!

Lykhoyda commented Jun 20, 2026

✅ Device verification complete (iOS + Android) — with one runtime finding

Finding (filed as B223 → will open a follow-up issue)

Uh oh!

Lykhoyda commented Jun 21, 2026

Update: Android hideKeyboard runner-routing added to this PR

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 21, 2026

Choose a reason for hiding this comment

Uh oh!

Lykhoyda commented Jun 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Update: Android `hideKeyboard` runner-routing added to this PR